Attention in Attention: Modeling Context Correlation for Efficient Video Classification

Attention mechanisms have significantly boosted the performance of video classification neural networks thanks to utilization perspective contexts. However, current research on attention generally focuses adopting a specific aspect contexts (e.g., channel, spatial/temporal, or global context) refine features and neglects their underlying correlation when computing attentions. This leads incomplete context hence bears weakness limited improvement. To tackle problem, this paper proposes an efficient attention-in-attention (AIA) method for element-wise feature refinement, which investigates feasibility inserting channel into spatio-temporal learning module, referred as CinST, also its reverse variant, STinC. Specifically, we instantiate dynamics aggregated along axis with average max pooling operations. The workflow AIA module is that first block uses one kind information guide gating weights calculation second targets at other context. Moreover, all computational operations in units act pooled dimension, results quite few cost increase (< 0.02%). verify our method, densely integrate it two classical network backbones conduct extensive experiments several standard benchmarks. source code available https://github.com/haoyanbin918/Attention-in-Attention .